Skip to content

PP-4: manifest version 2, row keys, per-file plain-stack confirmation - #110

Merged
guijacquemet merged 5 commits into
mainfrom
claude/pp-4-manifest-v2
Nov 1, 2026
Merged

PP-4: manifest version 2, row keys, per-file plain-stack confirmation#110
guijacquemet merged 5 commits into
mainfrom
claude/pp-4-manifest-v2

Conversation

@guijacquemet

@guijacquemet guijacquemet commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

This is PP-4 of the v0.4.0 plan (preprocessing-design-v0.4.md revision 4, §1, §3, §6 and §8). It is breaking: every v1 run is recomputed. It is based on main, with PP-0, PP-2 and PP-3 merged.

Commits: a move-only commit, the features, the X-before-Y follow-up, a merge of main, and a review fix.

Summary

Move only (commit 1):

  • review_axes, check_mapping and channel_count move from axes.py to axis_review.py; axes.py goes from 717 to 434 lines.
  • check_sources moves from manifest.py to sources.py. In commit 2, NarrowRange moves to narrow_ranges.py.
  • Baseline entries are re-keyed with tools/check.py --rename.

Features (commit 2):

  • Manifest version "2":
    • Each file's roles and axes_from (header or confirmed) are required; the reader and the writer both refuse an entry without them.
    • pixel_sizes per file: one entry per role X, Y, Z, T, each with value, unit and source. The source is header for now; PP-8 adds stated and pixels. An uncalibrated axis has value and source null, never a guessed value.
    • Masks gain size, mtime_ns, roles, axes_from and shared (mask_record, B3).
    • Optional conversion_record_sha256.
    • A version 1 manifest is still read, but no row from it is reused. The run reports "the earlier run was written before v0.4.0, without the axis check".
  • fingerprint.Fingerprint and fingerprint.RowKey give one rule for "the same file, read the same way" (audit D5). The key is the fingerprint, labels, axes, roles, axes_from, the read path and the row-changing settings.
    • run/entries.py builds the manifest entries and sorts files into new, changed and unchanged by row key.
    • This code moved out of run_experiment, which went from 195 to 172 executable lines.
  • axis_check.ConfirmedAxes (audit 2 B-1): a confirmation covers only the files it names, as they were when confirmed: fingerprint.confirm_axes records each file's Fingerprint, and run_experiment drops files changed since (review fix acc6fa7).
    • A plain stack outside it is the error plain_stack_not_confirmed, and is never read with roles confirmed for other files.
    • A plain AxisMapping still confirms every file the run finds.
  • X before Y (owner, follow-up e9108d2): a plain stack confirmed with X before Y (TXY) is the error axes_x_before_y; nothing is read transposed.
  • Docs: contract §2 and §9, a user guide message, plan §12 Q47, AGENTS.md.

Notes for review

  • Not wired in this PR:

    • the notebook does not yet pass the files it showed at confirmation (PP-10, per the plan);
    • the CLI's reuse of the recorded mapping without --axes (PP-7).

    For now, ConfirmedAxes is enforced through the API.

  • Read path: it is part of the row key (spec §1), so moving the experiment folder recomputes its rows. A test covers this.

  • Settings: settings that change every row are still reported once, as the rebuild reason. Row keys are compared with the current settings on both sides, so the new/changed/unchanged counts stay per file.

  • Test edits:

    • test_masks.py now checks only the mask path; the full mask entry is checked in test_manifest_v2.py.
    • The test-only plain() helper is shared as tiff_tree.write_plain.

Tests

  • New file tests/test_manifest_v2.py (12 tests):
    • version 2 roles, axes_from and pixel sizes; a plain stack recorded as confirmed and uncalibrated;
    • mask fingerprints;
    • the reader and writer refuse missing roles; recorded-size checks;
    • a v1 manifest gives no reused rows;
    • row reuse by row key (moving the experiment recomputes); the RowKey fields; Fingerprint;
    • per-file confirmation (plain_stack_not_confirmed names only the unconfirmed file); named images need no confirmation.
  • On old main the file fails at import: fingerprint and ConfirmedAxes do not exist there.
  • tests/test_x_before_y.py (9 tests): X-before-Y confirmations refused.
  • Review fix: test_a_plain_stack_changed_since_its_confirmation_is_not_covered fails without the fix.
  • Full gate as CI (ruff check, ruff format --check, tools/check.py --rules, pytest with .[dev,notebook,zarr]), at acc6fa7:
    • Python 3.13 with zarr 3: 1366 passed, 1 skipped (running as root ignores file modes).
    • Python 3.11 with zarr 2.18: 1348 passed, 19 skipped (that root skip plus 18 OME-NGFF 0.5 tests that need zarr 3).
    • CI green on all six jobs.

https://claude.ai/code/session_01WX6rdMV1r5jUPUR94kxV5a

Generated by Claude Code

claude added 2 commits November 1, 2026 16:20
@claude
PP-4 (1/3): move review_axes and check_sources into their own modules
31e3eb6 
Move only, so the manifest v2 and per-file confirmation work that follows
grows no oversized file:
- axis_review.py: check_mapping, review_axes, channel_count and their
  helpers (from axes.py, 717 -> 470 lines);
- sources.py: check_sources and its texts (from manifest.py, 732 -> 581
  lines).

Importers now use the new modules, and the package re-exports the same
names. The baseline entries are re-keyed with tools/check.py --rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WX6rdMV1r5jUPUR94kxV5a
@claude
PP-4 (2/2): manifest version 2, row keys, per-file plain-stack confir…
d9b109a 
…mation

Design v0.4 §1, §3, §6 (spec revision 4, PP-4). Breaking: every v1 run is
recomputed.

- Manifest version "2":
  - per file, roles and axes_from (header | confirmed) are required, by
    the reader and the writer; pixel_sizes are recorded by role with
    value, unit and source (recorded_sizes.RecordedSize; header for now);
  - per mask: size, mtime_ns, roles, axes_from and shared (mask_record,
    B3);
  - an optional conversion_record_sha256.
  A version 1 manifest is still read, but no row of it is reused ("the
  earlier run was written before v0.4.0, without the axis check").
- fingerprint.Fingerprint and fingerprint.RowKey are one rule for "the
  same file, read the same way" (audit D5): fingerprint, labels, axes,
  roles, axes_from, the read path, and the row-changing settings.
  run/entries.py builds the entries and sorts the files by row key, moved
  out of run_experiment, which shrinks.
- axis_check.ConfirmedAxes: a confirmation covers only the files it was
  given for (audit 2 B-1). A plain stack outside it is the error
  plain_stack_not_confirmed. A plain AxisMapping still confirms every file
  the run finds. The notebook passes its files in PP-10; the CLI's reuse
  of the recorded mapping comes in PP-7.
- NarrowRange moves to narrow_ranges.py so manifest.py stays under 600
  lines. Tests share tiff_tree.write_plain.

Docs: contract §2/§9, user guide message, plan §12 Q47, AGENTS.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WX6rdMV1r5jUPUR94kxV5a
@guijacquemet guijacquemet mentioned this pull request Nov 1, 2026
Merged
@guijacquemet
guijacquemet marked this pull request as ready for review November 1, 2026 16:59
@claude
PP-4 follow-up: refuse plain stacks confirmed X before Y; tests exemp…
e9108d2 
…t from annotations

- A plain stack whose confirmed roles put X before Y (TXY) is the error
  axes_x_before_y: nothing is ever read transposed. The message recommends
  saving again with Y before X, or OME-TIFF copies (owner decision).
- tools/check.py leaves tests out of the private-docstring rule, as Ruff
  already ignores ANN and D there (owner decision).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WX6rdMV1r5jUPUR94kxV5a
@guijacquemet
guijacquemet changed the base branch from claude/pp-3-split-run to main November 1, 2026 17:27
claude added 2 commits November 1, 2026 17:30
@claude
Merge main (PP-0, PP-2, PP-3 merged) into PP-4
eb1e863 
Conflicts: tools/check.py keeps main's noqa count over every source; the
baseline is main's keyed format, lowered to this branch's code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WX6rdMV1r5jUPUR94kxV5a
@claude
PP-4 review: a changed plain stack is not covered by its confirmation
acc6fa7 
Audit 2 B-1 and design v0.4 §3: a plain stack that is new *or changed*
must be confirmed again. ConfirmedAxes named files by path only, so a plain
stack rewritten under the same name stayed confirmed and was read with the
roles confirmed for its old contents.

- ConfirmedAxes.files now maps each file to its Fingerprint when it was
  confirmed; fingerprint.confirm_axes records it.
- run_experiment drops confirmed files whose fingerprint changed or that are
  gone (fingerprint.unchanged_confirmation), so such a plain stack is
  plain_stack_not_confirmed. The pure axis check stays free of I/O.
- run_experiment's counts come from the Comparison (161 executable lines,
  baseline lowered from 166).
- header_axis_issue's docstring no longer claims a header naming X before Y
  is refused: only confirmed roles of unnamed axes are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WX6rdMV1r5jUPUR94kxV5a
@guijacquemet Claude

guijacquemet commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Independent review of PP-4 (design v0.4 rev 4 §1, §3, §6, §9).

Finding, fixed in acc6fa7: ConfirmedAxes covered files by path only, so a plain stack rewritten under the same name stayed confirmed and was read with roles confirmed for its old contents (B-1 says new or changed). It now maps each file to its Fingerprint (fingerprint.confirm_axes), and run_experiment drops files changed or gone since (unchanged_confirmation), so they are plain_stack_not_confirmed. The new regression test fails without the fix. run_experiment went from 166 to 161 executable lines (baseline lowered). I also corrected a docstring that said a header naming X before Y is refused; only confirmed roles of unnamed axes are.

Verified:

  • Commit 31e3eb6 only moves code (git diff --color-moved).
  • Manifest v2: roles, axes_from and pixel_sizes by role with unit and source; uncalibrated axes are null and never guessed. Masks get a fingerprint and roles. The reader and writer refuse entries without roles. A v1 manifest gives no reused row and the run reports it.
  • axes_x_before_y (owner answer) is an error with a regression test, and nothing is read.
  • Reused rows were checked adversarially: changed roles (TYX→ZYX), bins, histogram range, robust quantiles, the range-check limit, labels (a renamed folder), the read path (a moved experiment) and the file fingerprint each recompute. missing_as_singleton is in the key through singleton_roles.
  • §9: the baseline diff, checked by hand, only removes or lowers entries. The two moved functions keep their entries under their new names.
  • Gate: ruff, format and tools/check.py --rules pass. Python 3.13 with zarr 3: 1366 passed, 1 skipped. Python 3.11 with zarr 2.18: 1348 passed, 19 skipped. All six CI jobs are green.

For later, not blocking:

  • planes._finish still transposes a plane when a header names X before Y. Only plain-stack confirmations are refused. The owner should decide whether named X-before-Y headers are refused too (PP-6 read_plane must never transpose).
  • check_sources still accepts v1 manifests; RunInputs (PP-6) refuses them.

Generated by Claude Code

@guijacquemet
guijacquemet merged commit fb5f853 into main Nov 1, 2026
7 checks passed
@guijacquemet guijacquemet mentioned this pull request Nov 1, 2026
Merged
@guijacquemet
guijacquemet deleted the claude/pp-4-manifest-v2 branch November 14, 2026 08:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@guijacquemet @claude